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(57) An image search apparatus acquires associa- 
tive words in relation to an input query word, and makes 
a keyword search of image information on the basis of 
the obtained associative words and input query word. 
Parallel to the keyword search, the apparatus acquires 
a query word and sensory patterns corresponding to the 



query word, and makes a feature amount search using 
the feature amounts of the obtained sensory patterns. 
A search result integration process integrates these 
search results at weight ratios which are set in advance 
to obtain final search results. In this manner, the multi- 
media information wanted can be accurately extracted 
with respect to the input query word. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to an information 

search apparatus and method for searching information 
on the basis of an input query word. IVIore specifically, 
the present invention relates to an information search 
apparatus and method for managing a plurality of kinds 
of multimedia information, and searching the managed 
multimedia information for desired multimedia informa- 
tion, and a computer readable memory. 
[0002] A conventional information search apparatus, 
which searches multimedia information, e.g., image in- 
formation, makes a search using data (keywords) de- 
rived from subjective evaluation results of one or a plu- 
rality of persons for test images, physical image features 
extracted from images, and the like. 
[0003] When an image is searched for using a key- 
word, a required image is obtained by matching a given 
keyword with that corresponding to the image. Also, a 
scheme for obtaining an image, that cannot be obtained 
by full-word matching with an input keyword, by match- 
ing not only the input keyword but also an associated 
keyword associated with the input keyword with a key- 
word corresponding to an image, is proposed. Further- 
more, a search scheme which obtains an image with 
similar color information by detecting a correspondence 
between the input keyword and color information using, 
e.g., color information of images is proposed. 
[0004] In the image search using keywords, an im- 
pression that a person receives upon watching an im- 
age, or key information linked with the impression is ap- 
pended to image information and is used in search. As 
the key information, words that express impressions 
evoked by images such as "warm", "cold", and the like, 
and words that represent objects in drawn images such 
as "kitty", "sea", "mountain", and the like are appended 
as keywords. Also, local image feature components on 
drawn images are subjectively evaluated and are often 
appended as key information. For example, information 
that pertains to a color such as "red", "blue", and the 
like, information that pertains to a shape such as 
"round", "triangular, "sharp", and the like, and informa- 
tion that pertains to a texture such as "sandy", "smooth", 
and the like are expressed using words and icons, are 
appended to images as key information, and are used 
in search. 

[0005] In a system in which physical image feature 
amounts are extracted from images, and are used in im- 
age search, image features include local colors painted 
on images, overall color tones, and shapes, composi- 
tions, textures, and the like of objects on drawn images. 
An image feature amount is extracted from segmented 
regions or blocks obtained by segmenting the overall im- 
age into regions based on color information, or seg- 
menting the image into blocks each having a given area, 
or is extracted from the entire image. Physical image 



features include, e.g., color information, density distri- 
bution, texture, edge, region, area, position, frequency 
distribution, and the like of an image. 
[0006] However, in the above search scheme, when 

s an image including a keyword that matches the input 
query word is searched for, images which do not match 
the search request of the searcher are often obtained. 
Especially, when an image search is made using an ab- 
stract query word such as a "refreshing" image, images 

10 found by the search are limited. To solve this problem, 
a search may be made by unfolding the query word "re- 
freshing" to keywords which are associated with that 
query word. However, when such scheme is used, im- 
ages which are not "refreshing" may be mixed in search 

15 results. 

[0007] In this manner, the operator cannot designate 
query conditions for obtaining a desired search result 
with respect to a search request indicated by the input 
keyword, and cannot obtain an intended search result. 
20 For example, even when the operator wants to find only 
images having "refreshing" feature patterns with respect 
to a search request "refreshing", images having content 
words associated from the search request "refreshing" 
such as a music score of a "refreshing" music, a "re- 
25 freshing" athlete", and the like, are presented, i.e., im- 
ages which do not match the search request are pre- 
sented. 

[0008] In place of a query word, a query image may 
be input, and a search may be made using its feature 
30 amount. However, in this case, a query image which re- 
flects the searcher's will must be prepared, and it is dif- 
ficult to select a query image, resulting in poor operabil- 
ity. 



[0009] The present invention has been made in con- 
sideration of the above-mentioned problems, and has 

as its object to provide an image search method and 
apparatus which can extract the information wanted with 
high precision with respect to an input query word. 
[001 0] In order to achieve the above object, according 
to one aspect of the present invention, there is provided 
an information search apparatus for searching informa- 
tion based on an input query word, comprising first 
search means for determining a query keyword on the 
basis of the query word, and searching information on 
the basis of the query keyword, second search means 
for determining a feature amount corresponding to the 
query word, and searching information on the basis of 
the feature amount, setting means for setting a search 
weight to be assigned to search results of the first and 
second search means, and integration means for inte- 
grating search results obtained by the first and second 
search means in accordance with the search weight set 
by the setting means. 

[0011] In order to achieve the above object, according 
to another aspect of the present invention, there is pro- 
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vided an information search method tor searching infor- 
mation based on an input query word, comprising the 
first search step of determining a query keyword on the 
basis of the query word, and searching information on 
the basis of the query keyword, the second search step 
of determining a feature amount corresponding to the 
query word, and searching information on the basis of 
the feature amount, the setting step of setting a search 
weight to be assigned to search results in the first and 
second search steps, and the integration step of inte- 
grating search results obtained in the first and second 
search steps in accordance with the search weight set 
in the setting step. 

[001 2] Other features and advantages of the present 
invention will be apparent from the following description 

taken in conjunction with the accompanying drawings, 
in which like reference characters designate the same 
or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] The accompanying drawings, which are incor- 
porated in and constitute a part of the specification, il- 
lustrate embodiments of the invention and, together with 
the description, serve to explain the principles of the in- 
vention 

Fig. 1 is a block diagram showing an example of the 
arrangement of an information processing appara- 
tus which constructs an image search apparatus 
according to an embodiment of the present inven- 
tion; 

Fig. 2 is a block diagram depicting the processing 
arrangement in the image search apparatus ac- 
cording to the embodiment of the present invention; 
Fig. 3 is a view showing a display example of search 
perspectives in association with a search request 
word input in a search request input process 201; 
Fig. 4 is a view showing a display example of a 
weight designation control panel for designating a 
search weight for a search using associative words, 
and a search weight for a search using sensory pat- 
terns, in the search request word input in a search 
request input process 201; 

Fig. 5 is a table showing the data structure of an 
image holding unit 218 which stores image IDs in 
correspondence with image file storage paths; 
Fig. 6 is a table showing an example of the data 
structure of an image content word holding unit 219 
which stores image IDs in correspondence with im- 
age content words; 

Fig. 7 is a table which stores data of the image con- 
tent word holding unit shown in Fig. 6 as a list of 
image IDs using image content words as keys: 
Fig. 8 Is a table showing an example of the data 
structure of a concept discrimination dictionary 205; 
Fig. 9 is a table showing an example of the data 
structure of an associative word dictionary 211 ; 



Fig. 1 0 is a table for explaining the data holding for- 
mat in a search result holding unit 216; 
Fig. 11 is a table for explaining an example of the 
data structure of an unfolded sensory pattern hold- 

s ing unit 213 shown in Fig. 2; 

Fig. 12 is a table showing an example of the data 
structure of an image word/sensory pattern corre- 
spondence holding unit 215 shown in Fig. 2; 
Fig. 13 is a table showing the data structure of a 

10 sensory pattern holding unit 220 shown in Fig. 2; 

Fig. 1 4 is a table showing a data example obtained 
upon extracting Image feature amounts from a sin- 
gle image by an image feature amount extraction 
process; 

15 Fig. 1 5 is a table showing an example of image fea- 
ture amounts in this embodiment, which are ob- 
tained by extracting representative colors in units of 
image regions/blocks; 

Fig. 1 6 is a table showing a storage example of an 
20 image feature amount holding unit 222 shown in 
Fig. 2; 

Fig. 17 is a table showing a data storage example 
of an image feature amount/sensory pattern corre- 
spondence holding unit 223 shown in Fig. 2; 
25 Fig. 18 is a flow chart for explaining the operation 
of the present invention; 

Fig. 19 Is a flow chart showing the details of the 
search request input process 201 (step SI in Fig. 
18); 

30 Fig. 20 is a flow chart showing the details of an as- 
sociative word unfolding process 208 and an image 
content word search process 210 using associative 
words (step S4 in Fig. 18); 
Fig. 21 is a flow chart showing the details of a sen- 
35 sory pattern unfolding process 21 2, sensory pattern 
search process 214, and search result integration 
process 217; and 

Fig. 22 is a flow chart showing an example of a pre- 
process of a search, which is done upon registering 
40 an image. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

45 [0014] Preferred embodiments of the present Inven- 
tion will now be described in detail in accordance with 
the accompanying drawings. 

[0015] Fig. 1 shows an example of the arrangement 

of an information processing apparatus which con- 
50 structs an image search apparatus according to this em- 
bodiment. 

[0016] Referring to Fig. 1, reference numeral 11 de- 
notes a microprocessor (to be referred to as a CPU 
hereinafter), which makes computations, logical deci- 
55 sions, and the like for Image Information search in ac- 
cordance with control programs, and controls individual 
building components connected to an address bus AB, 
control bus CB, and data bus DB via these buses. The 
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address bus AB transfers an address signal indicating 
the building component to be controlled by the CPU 11 . 
The control bus CB transfers a control signal for each 
building component to be controlled by the CPU 1 1 . The 
data bus DB transfers data among the respective build- 
ing components. 

[0017] Reference numeral 12 denotes a read-only 
memory (to be referred to as a ROM hereinafter), which 
stores a boot processing program and the like executed 
by the CPU 11 upon starting up the apparatus of this 
embodiment. Reference numeral 13 denotes a rewrita- 
ble random access memory (to be referred to as a RAM 
hereinafter) which is configured by 16 bits per word, and 
is used as a temporary storage of various data from the 
respective building components. Especially, in this em- 
bodiment, the RAM 13 provides various data holding 
units such as a query word holding unit 202, search per- 
spective holding unit 203, search weight holding unit 
204, determined weight holding unit 207, unfolded as- 
sociative word holding unit 209, unfolded sensory pat- 
tern holding unit 21 3, and search result holding unit 216, 
which will be described later with reference to Fig. 2. 
[0018] Reference numeral 14 denotes an external 
memory (DISK), which stores a concept discrimination 
dictionary 205 and associative word dictionary 211 , and 
provides data holding units such as an image word/sen- 
sory pattern correspondence holding unit 215, image 
content word holding unit 219, image holding unit 218, 
sensory pattern holding unit 220, image feature amount 
holding unit 222, and image feature amount/sensory 
pattern correspondence holding unit 223, which will be 
described later with reference to Fig. 2. As a storage 
medium of the external memory 14, a ROM, floppy disk, 
CD-ROM, memory card, magnetoopticaldisk, or the like 
can be used. 

[001 9] Also, the external memory 1 4 stores programs 
for respectively implementing the respective processes, 
i.e., a search request input process 201, weight deter- 
mination process 206, associative word unfolding proc- 
ess 208, image content word search unit 210 using as- 
sociative words, sensory pattern unfolding process 212, 
sensory pattern search process 214, search result inte- 
gration process 217, image feature amount extraction 
process 221, and sensory pattern determination proc- 
ess 224, which will be described later with reference to 
Fig. 2 and the like. These programs are loaded onto the 
RAM 13 as needed, and are executed by the CPU 11. 
[0020] Reference numeral 15 denotes a keyboard 
(KB) which has alphabet keys, hiragana keys, katakana 
keys, character symbol input keys for inputting a period, 
comma, and the like, a search key for instructing a 
search (a function key on a general keyboard may be 
used instead), and various function keys such as cursor 
moving keys for instructing cursor movement, and the 
like. Also, a pointing device such as a mouse or the like 
(not shown) may be connected. 
[0021] Reference numeral 16 denotes a display video 
memory (to be referred to as VRAM hereinafter) for stor- 



ing a pattern of data to be displayed. Reference numeral 
17 denotes a CRT controller (to be referred to as a 
CRTC hereinafter) for displaying the contents stored in 
the VRAM 16 on a CRT 18. Reference numeral 18 de- 
s notes a display device (CRT) using, e.g., a cathode ray 
tube, or the like. The dot display pattern and cursor dis- 
play on the CRT 18 are controlled by the CRTC 17. Note 
that various other displays such as a liquid crystal dis- 
play, and the like may be used as the display device. 
Reference numeral 19 denotes a network controller 
(NIC), which connects the image search apparatus of 
this embodiment to a network such as Ethernet or the 
like. 

[0022] The image search apparatus of this embodi- 
ment constructed by the aforementioned building com- 
ponents operates in accordance with various inputs 
from the keyboard 15 and various inputs supplied from 
the network controller via the network. Upon receiving 
the input from the keyboard 15 or network controller 1 9, 
an interrupt signal is sent to the CPU 11 . Upon receiving 
the interrupt signal, the CPU 11 reads out various control 
data stored in the external memory 14, and executes 
various kinds of control in accordance with these control 
data. Also, the present invention is achieved by supply- 
ing a storage medium that stores a program according 
to the present invention to a system or apparatus, and 
by reading out and executing program codes stored in 
the storage medium by a computer of the system or ap- 
paratus. 

[0023] Fig. 2 is a block diagram depicting the process- 
ing arrangement in the image search apparatus of this 
embodiment. 

[0024] Referring to Fig. 2, reference numeral 201 de- 
notes a search request input process for inputting query 
items (query word, search perspective or category, and 
search weight in this embodiment; to be described in 
detail later) that pertain to the information wanted. Ref- 
erence numeral 202 denotes a query word holding unit 
for storing a query word input by the search request in- 
put process 201. Reference numeral 203 denotes a 
search perspective holding unit for storing a search per- 
spective input by the search request input process 201 . 
Reference numeral 204 denotes a search weight hold- 
ing unit for storing a search weight input by the search 
request input process 201 . 

[0025] Reference numeral 205 denotes a concept dis- 
crimination dictionary having a search perspective that 
pertains to a concept as information wanted, an antithet- 
ic concept having a meaning contrary or antonymous to 
the concept as the information wanted, and two kinds of 
coefficients (associated weight and sensory pattern 
weight) for weight discrimination upon searching for a 
concept, as shown in Fig. 8. Note that the concept dis- 
crimination dictionary 205 will be described in detail later 
with reference to Fig. 8. Reference numeral 206 denotes 
a weight determination process for giving weights (as- 
sociated weight and sensory pattern weight) indicating 
the weight balance on associative words (obtained by 
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an associative word unfolding processing unit 208) and 
sensory patterns (obtained by a sensory pattern unfold- 
ing processing unit 212) in actual search using a query 
word stored in the query word holding unit 202. Refer- 
ence numeral 207 denotes a determined weight holding 
unit for holding the search weight determined by the 
weight determination process 206. 
[0026] Reference numeral 208 denotes an associa- 
tive word unfolding process for unfolding the query word 
stored in the query word holding unit 202 into associa- 
tive words with reference to an associative word diction- 
ary 211, obtaining an antithetic concept antonymous to 
that query word from the concept discrimination diction- 
ary 205, and unfolding the obtained antithetic concept 
into associative words with reference to the associative 
word dictionary 211 . Reference numeral 209 denotes an 
unfolded associative word holding unit for holding the 
associative words (including those of the antithetic con- 
cept) unfolded by the associative word unfolding proc- 
ess 208. Reference numeral 21 0 denotes an image con- 
tent word search process using associative words (to 
be referred to as an image content word search process 
hereinafter), which finds image content words, that are 
stored in an image content word holding unit 219 and 
match the unfolded associative words, by search with 
reference to the unfolded associative word holding unit 
209. Reference numeral 211 denotes an associative 
word dictionary for storing associative words in units of 
concepts serving as index words in correspondence 
with associative perspectives (this process will be de- 
scribed in more detail later with reference to Fig. 9). 
[0027] Reference numeral 212 denotes a sensory 
pattern unfolding process for unfolding the query word 
stored in the query word holding unit 202 into sensory 
patterns with reference to an image word/sensory pat- 
tern correspondence holding unit 215, obtaining an an- 
tithetic concept antonymous to that query word from the 
concept discrimination dictionary 205, and unfolding the 
obtained antithetic concept into sensory patterns with 
reference to the image word/sensory pattern corre- 
spondence holding unit 215. 

[0028] Reference numeral 213 denotes an unfolded 
sensory pattern holding unit for temporarily storing the 
sensory patterns unfolded by the sensory pattern un- 
folding process 21 2. Storage of data in the unfolded sen- 
sory pattern holding unit 21 3 will be described later with 
reference to Fig. 11 . Reference numeral 214 denotes a 
sensory pattern search process for finding sensory pat- 
terns, which are stored in the sensory pattern holding 
unit 220 and are similar to the unfolded sensory pat- 
terns, by search with reference to the sensory pattern 
holding unit 220. 

[0029] Reference numeral 215 denotes an image 

word/sensory pattern correspondence holding unit for 
storing the correspondence between the image words 
and sensory patterns by storing sensory pattern I Ds cor- 
responding to sets of image words and associative 
words associated with the image words. Note that the 



image word/sensory pattern correspondence holding 
unit 215 will be described in detail later with reference 
to Fig. 12. 

[0030] Reference numeral 216 denotes a search re- 
s suit holding unit for storing image IDs found by searches 
of the image content word search process 210 and sen- 
sory pattern search process 214. Reference numeral 
217 denotes a search result integration process for in- 
tegrating the search results of image content words us- 
10 ing the associative words, and the search results of sen- 
sory patterns stored in the search result holding unit 
216, on the basis of the search weights obtained by the 
weight determination process 206 and stored in the de- 
termined weight holding unit 207. 
15 [0031] Reference numeral 218 denotes an image 
holding unit for storing image information serving as test 
images. Reference numeral 219 denotes an image con- 
tent word holding unit for verbalizing and storing con- 
cepts appended to images stored in the image holding 
20 unit 218 to express their contents. Reference numeral 
220 denotes a sensory pattern holding unit for storing 
matching levels between image information of the indi- 
vidual images stored in the image holding unit 218 and 
sensory patterns. More specifically, the sensory pattern 
25 holding unit 220 stores the matching levels with sensory 
patterns in units of image IDs. The data contents of the 
sensory pattern holding unit 220 will be described later 
with reference to Fig. 1 3. 

[0032] Reference numeral 221 denotes an image fea- 
30 ture extraction process for extracting physical image 
feature amounts from image information stored in the 
image holding unit 21 8. Physical image feature amounts 
are visual features or signatures extracted from regions 
segmented on the basis of color information, blocks 
35 each segmented to have a given area, or the entire im- 
age. For example, the image feature is numerical infor- 
mation such as the color distribution or histogram, den- 
sity distribution, texture, edge, frequency distribution, 
and the like of an image. Note that the image feature 
40 amounts will be described in detail later with reference 
to Fig. 14. 

[0033] Reference numeral 222 denotes an image fea- 
ture amount holding unit for storing the image feature 
amounts obtained by the image feature amount extrac- 
ts tion process 221. Reference numeral 223 denotes an 
image feature amount/sensory pattern correspondence 
holding unit for storing image feature amounts and sen- 
sory patterns in correspondence with each other. The 
image feature amount/sensory pattern corresponding 
50 holding unit 223 stores sensory pattern IDs and image 
feature amount data corresponding to those IDs. Note 
that the data structure of the image feature amount/sen- 
sory pattern correspondence holding unit 223 will be de- 
scribed in detail later with reference to Fig. 17. 
55 [0034] Reference numeral 224 denotes a sensory 
pattern determination process for comparing image fea- 
ture amounts extracted from a sensory pattern and im- 
age information and determining their matching level 
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with reference to the image feature amount holding unit 
222 and image feature amount/sensory pattern corre- 
spondence holding unit 223, and registering the result 
in the aforementioned sensory pattern holding unit 220. 
[0035] A display example of a search perspective that 
pertains to search request items input at the search re- 
quest input processing unit 201 will be explained below 
with reference to Fig. 3. 

[0036] Fig. 3 shows a display example of search per- 
spectives in association with the query word input by the 
search request input process 201 . When a query word 
is input by operating, e.g., the keyboard 1 5, the concept 
discrimination dictionary 205 shown in Fig. 8 is searched 
using the query word as an index word to extract corre- 
sponding search perspectives. Fig. 3 illustrates that 
three search perspectives "color tone", "taste", and 
"general atmosphere" are available in relation to a query 
word "mild", and hatched "color tone" is selected as the 
search perspective. When the user presses (clicks) an 
OK button in this state, the search perspective "color 
tone" is selected, and is held in the search perspective 
holding unit 203. 

[0037] By pressing one of the cursor moving keys on 
the keyboard 15, the hatching moves from "color tone" 
to "taste" or "general atmosphere", and the user can 
designate a desired search perspective or category. 
Note that "mild" as the query word is held in the query 
word holding unit 202, and the selected search perspec- 
tive ("color tone" in Fig. 3) is held in the search perspec- 
tive holding unit 203. 

[0038] A display example on the control panel when 
the operator instructs the search weight balance on a 
search using associative words and a search using sen- 
sory patterns in actual search will be explained below 
with reference to Fig. 4. 

[0039] Fig. 4 shows a display example of a weight 
designation control panel for instructing search weights 
for searches using associative words and sensory pat- 
terns in the search request input process 201. As de- 
scribed above, in this embodiment, a search using as- 
sociative words and a search using the feature amounts 
of images (sensory patterns) based on the query word 
are made, and the search results are integrated. In this 
integration process, the two search results are weight- 
ed. On the weight designation control panel, the user 
can designate a search weight for a search using asso- 
ciative words, and that for a search using sensory pat- 
terns. That is, the user can designate the weight balance 
on a search using associative words or that using sen- 
sory patterns in actual search. 

[0040] Referring to Fig. 4, when the user slides a slide 
button 41 to the left, an instruction that sets a heavier 
weight on a search using associative words is issued; 
when he or she slides the slide button 41 to the right, an 
instruction that sets a heavier weight on a search using 
sensory patterns is issued. When the user designates 
search weights using the slide button 41 and then press- 
es an OK button 43, a search weight instruction is is- 



sued. Note that a button 42 is pressed when no search 
weights are clearly designated. Upon depression of the 
button 42, predetermined weight values (which are ob- 
tained from an associated weight 83 and sensory pat- 

s tern weight 84 in the concept discrimination dictionary 
205 (Fig. 8)) are used. The set weights are stored in the 
search weight holding unit 204. Note that the buttons 41 
to 43 on the control panel may be clicked by a pointing 
device (not shown). 

10 [0041] The structure of the image holding unit 218 will 
be described below using Fig. 5. 
[0042] Fig. 5 shows the data structure of the image 
holding unit 218 which stores image IDs in correspond- 
ence with image file storage paths. Referring to Fig. 5, 

15 reference numeral 51 denotes an image ID which is an 
identification number uniquely assigned to one image 
file in this image database. Reference numeral 52 de- 
notes a file path which indicates the storage location of 
an image file corresponding to the image ID in the ex- 

20 ternal memory 1 4, and corresponds to the directory and 
file of MS-DOS. 

[0043] An image file is divided into header and image 
data fields (not shown in Fig. 5). The header field stores 
information required for reading data from that image 
25 file, and additional information that explains the image 
contents. As such information, an image format identi- 
fier indicating the image format name of the image, file 
size, image width, height, and depth, the presence/ab- 
sence of compression, color pallet information, resolu- 
te tion, offset to the storage location of image data, and the 
like are stored. The image data field stores image data 
in turn. This embodiment uses the BMP format of Micro- 
soft Corp. as such image format, but other compression 
formats such as GIF, JPEG, FlashPix, and the like may 
55 be used. 

[0044] The structure of the image content word hold- 
ing unit 21 9 will be described below with the aid of Fig. 6. 
[0045] Fig. 6 shows an example of the data structure 
of the image content word holding unit 21 9 which stores 
40 image I Ds in correspondence with image content words. 
Referring to Fig. 6, reference numeral 61 denotes an 
image ID, which corresponds to the image ID 51 in Fig. 
5. Reference numeral 62 denotes an image content 
word which stores a word for expressing each image 
45 identified by the image ID 61 . The image content word 
is a keyword which verbally expresses an image feature 
in an image, and is stored as a character code (e.g., 
Unicode). A plurality of keywords may be stored per im- 
age, and the image content word holding unit 21 9 is con- 
50 structed as a list of image content words 62 using image 
IDs 61 as keys. 

[0046] Fig. 7 shows a table which stores the data of 
the image content word holding unit shown in Fig. 6 as 
a list of image IDs using image content words as keys. 
55 Referring to Fig. 7, all image IDs that contain image con- 
tent words 71 as keywords are stored as image IDs 72. 
[0047] The structure of the concept discrimination dic- 
tionary 205 will be described below using Fig. 8. 
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[0048] Fig. 8 shows an example of the data structure 
of the concept discrimination dictionary 205. The con- 
cept discrimination dictionary 205 provides information 
that pertains to a query word serving as a search re- 
quest, and is a table which stores index words 80 cor- 
responding to query words, search perspectives 81 as- 
sociated with index words, antithetic concepts 82 having 
meanings contrary to the index words in units of search 
perspectives, associative word weights 83 used upon 
searching the index words, and sensory pattern weights 
84 used upon searching the index words in correspond- 
ence with each other. 

[0049] The structure of the associative word diction- 
ary 21 1 will be explained below with reference to Fig. 9. 
[0050] Fig. 9 shows an example of the data structure 
of the associative word dictionary 211 . The associative 
word dictionary 211 is composed of associative IDs 90 
each of which assigns a unique number to a set of as- 
sociative words for each index word, index words 91 
each serving as a start point of association, associative 
words 92 evoked by the index words 91 , associative per- 
spectives 93 which are relevant to associations of the 
associative words 92, and association strengths 94 
each indicating the strength of association between 
each pair of index word 91 and associative word 92. 
[0051] In this embodiment, the association strength 
94 assumes an absolute value ranging from 0 to 1 0, and 
its sign indicates direction of association of the associ- 
ative word. More specifically, when the association 
strength is a positive value, it indicates a stronger asso- 
ciative relationship (higher bilateral association) as the 
association strength value is larger; when the associa- 
tion strength is a negative value, it indicates a harder 
associative relationship as the association strength val- 
ue is larger. For example, an associative word "folkcraft 
article" corresponding to an index word "simple" in as- 
sociative data with the associative ID = 126533 can be 
associated with strength "6", but an associative word 
"chandelier" in associative data with the associative ID 
= 126536 is hardly associated with strength "9" since its 
association strength is a negative value. 
[0052] The structure of the search result holding unit 
216 will be described below with reference to Fig. 1 0. 
[0053] Fig. 10 shows the data holding format in the 
search result holding unit 216. As described above, the 
search result holding unit 216 stores image IDs which 
are found by searches of the image content word search 
process 210 using associative words and the sensory 
pattern search process 214. 

[0054] Referring to Fig. 1 0, reference numeral 1 00 de- 
notes a field for storing image IDs found by search; 101, 
a field for storing the number of matched associative 
words with positive association strengths by the image 
content word search process 217. An associative word 
I D list 1 02 stores a maximum of 20 associative word I Ds 
90 (Fig. 9) of matched associative words in the associ- 
ative word dictionary 211. When the number 101 of 
matched associative words is zero, the associative ID 



102 is filled with NULL code. Reference numeral 103 
denotes a field for storing associative matching levels 
of associative words with respect to the image IDs 100. 
When the number 101 of matched associative words is 

s zero, the association matching level 1 03 also stores ze- 
ro. 

[0055] Reference numeral 104 denotes a field for 
storing the number of sensory patterns with highest sim- 
ilarity, which are found by search by the sensory pattern 

10 search process 223. Reference numeral 1 05 denotes a 
sensory pattern ID list which stores a maximum of 20 
sensory pattern IDs of matched sensory patterns. When 
the number 104 of matched sensory patterns is zero, 
the sensory pattern ID 105 is filled with NULL code. Ref- 

15 erence numeral 106 denotes a field for storing the 
search matching level of a sensory pattern search with 
respect to the image ID 100. When the number 104 of 
matched sensory patterns is zero, the sensory pattern 
matching level 106 stores zero. Reference numeral 107 

20 denotes a field for storing an integrated matching level 
(obtained by the search result integration process 217) 
of the image ID 100 with respect to the search request, 
which is calculated using the associative matching level 

103 and sensory pattern matching level 106 as param- 
25 eters. 

[0056] The structure of the above-mentioned unfold- 
ed sensory pattern holding unit 21 3 will be described in 
detail below with reference to Fig. 11. 
[0057] Fig. 11 is a table for explaining an example of 

30 the data structure of the unfolded sensory pattern hold- 
ing unit 21 3 shown in Fig. 2. Referring to Fig. 1 1 , refer- 
ence numeral 1 1 0 denotes an image word as an unfold- 
ing source from which this sensory pattern has evolved 
upon unfolding, and the same image word as that in the 

55 query word holding unit 202 is stored. In this embodi- 
ment, a character string "refreshing" is stored, and ends 
with NULL code. Reference numeral 111 denotes the 
number of sensory patterns obtained by unfolding the 
image word 110 with reference to the image word/sen- 

40 sory pattern correspondence holding unit 215. For ex- 
ample, when the contents of the image word/sensory 
pattern correspondence holding unit 215 are as shown 
in Fig. 1 2, the number of sensory patterns unfolded from 
the image word "refreshing" is 7. Reference numeral 1 1 2 

45 denotes an address indicating the storage location area 
of data obtained by actually unfolding the image word 
"refreshing". In the example shown in Fig. 11, the stor- 
age location address 112 is linked with unfolded data 
115 shown in Fig. 11 . 

50 [0058] In the data 115, data actually unfolded from "re- 
freshing", i.e., sets of associative words and sensory 
patterns corresponding to the number 111 of sensory 
patterns are stored. In this embodiment, seven sets of 
associative words and sensory patterns are stored. For 

55 example, an associative word 114 is that of the image 
word "refreshing" and stores a character string "forest" 
in this example. Also, a sensory pattern ID 113 corre- 
sponds to the image word "refreshing" and its associa- 
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live word "forest", and stores "5" in this example. The 
same applies to other sets of associative words and sen- 
sory patterns. 

[0059] The structure of the aforementioned image 
word/sensory pattern correspondence holding unit 215 
will be described in detail below using Fig. 12. 
[0060] Fig. 1 2 shows an example of the data structure 
of the image word/sensory pattern correspondence 
holding unit 21 5 in Fig. 2. Referring to Fig. 1 2, reference 
numeral 120 denotes an image word serving as an un- 
folding source of this sensory pattern. In Fig. 12, char- 
acter strings "refreshing", "tropical'V and the like are 
stored, and end with NULL code. Reference numeral 
121 denotes an associative word unfolded from each 
image word 120. In this embodiment, associative words 
"forest", "tableland", "blue sky", and the like are stored 
in correspondence with "refreshing", and these charac- 
ter strings end with NULL code. When no character 
string is stored in this field 121, i.e., NULL code alone 
is stored, this sensory pattern applies to all image words 
"refreshing"; no specific associative word has been des- 
ignated. 

[0061] Reference numeral 122 denotes a sensory 
pattern ID corresponding to the image word 1 20 and as- 
sociative word 1 21 . In this embodiment, "005" and "006" 
are stored as sensory pattern IDs corresponding to the 
image word "refreshing" and its associative word "for- 
est". Also, sensory patterns for "not refreshing" as an 
antithetic concept of "refreshing" are stored. In this em- 
bodiment, for "not refreshing", no associative words are 
registered and "001" and "010" are registered as sen- 
sory pattern IDs. 

[0062] The structure of the above-mentioned sensory 
pattern holding unit 220 will be described in detail below 
using Fig. 13. 

[0063] Fig. 1 3 shows the data structure of the sensory 
pattern holding unit 220 in Fig. 2. Referring to Fig. 13, 
reference numeral 131 denotes an image ID for identi- 
fying an image to be registered. The image IDs 131 use 
the same ones as those stored in the image holding unit 
218, and uniquely define images in this system. A field 
132 stores sensory pattern IDs. In this embodiment, 
since the matching levels between each image and all 
sensory patterns stored in the image feature amount/ 
sensory pattern correspondence holding unit 223 are 
calculated, all the sensory pattern IDs (1 to m) are 
stored. Reference numeral 133 denotes a numerical 
value indicating the matching level between each image 
and sensory pattern. The matching level assumes a val- 
ue ranging from 0 to 1 ; 0 indicates the image does not 
match the sensory pattern at all, and the matching level 
becomes higher as it is closer to 1 . For example, in this 
embodiment, the matching level between image with the 
image ID - 001 and sensory pattern 1 is 0.10, and the 
matching level between that image and sensory pattern 
2 isO. 

[0064] The aforementioned image feature amounts 
will be explained in detail below with reference to Fig. 1 4. 



[0065] Fig. 14 shows a data example obtained upon 
extracting the image feature amounts from one image 
by the image feature amount extraction process. In Fig. 
14, XI, X2, X3,..., Xn represent image features, B1, 

s B2,..., Bm represent regions/blocks from which image 
feature amounts are extracted, and x11 to xmn repre- 
sent image feature amounts extracted from the individ- 
ual regions/blocks. That is, feature amounts that pertain 
to physical image features XI to Xn are obtained in units 

10 of regions/blocks. 

[0066] Fig. 15 exemplifies a case wherein chromatic 
image feature amounts are especially extracted. In this 
case, representative colors are extracted in units of re- 
gions or blocks of an image. Referring to Fig. 1 5, a phys- 

15 ical image feature is expressed by "representative color" 
and "feature amount", representative colors extracted 
from regions/blocks 81, 82,..., Bn are C1(R1, G1, 81), 
C2(R2, G2, B2),..., Cn(Rn, Gn, Bn), and their image fea- 
ture amounts are c1 to cn. 

20 [0067] The structure of the image feature amount 
holding unit 222 will be described below using Fig. 16. 
[0068] Fig. 1 6 shows a storage example of the image 
feature amount holding unit 222 in Fig. 2. Referring to 
Fig. 16, reference numeral 161 denotes an image ID for 

25 identifying an image to be registered. The image I Ds 1 61 
use the same ones as those stored in the image holding 
unit 218. Reference numeral 162 denotes a block or re- 
gion number from which an image feature amount is ex- 
tracted. In this embodiment, B1 , B2,.., Bm represent the 

30 region/block numbers. Reference numeral 1 63 denotes 
information (in this embodiment, a representative color 
is used) indicating an image feature extracted from each 
region or block (B1, B2,..., Bm). This embodiment ex- 
emplifies a case wherein chromatic image features are 

55 extracted, and for example, a plurality of pieces of infor- 
mation C11 (R11, Gil, 811),..., Gn1(Rn1, Gnl, 8n1) in- 
dicating colors are stored. Reference numeral 164 de- 
notes image feature amounts of image features extract- 
ed from the individual regions/blocks. In this embodi- 

40 ment, c11 cnl are stored as the feature amounts of 
Oil (R11, Gil, 811),..., Gn1(Rn1, Gnl, Bnl). 
[0069] The structure of the image feature amount/ 
sensory pattern correspondence holding unit 223 will be 
described in detail below using Fig. 17. 

45 [0070] Fig. 17 shows a data storage example of the 
image feature amount/sensory pattern correspondence 
holding unit 223 in Fig. 2. Referring to Fig. 17, reference 
numeral 171 denotes a sensory pattern ID, which 
uniquely identifies a sensory pattern. Reference numer- 

50 al 1 72 denotes image feature amount data correspond- 
ing to each sensory pattern ID. In this embodiment, a 
sensory pattern is expressed by a chromatic image fea- 
ture amount, and a combination of color components 
(values in a color space such as RGB, HVG, or the like) 

55 corresponding to each sensory pattern ID is stored. In 
this embodiment, RGB values assume integers ranging 
from 0 to 255. A maximum of m colors correspond to 
each sensory pattern ID. 
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[0071] The sensory pattern determination process 
224 calculates the matching levels between each of im- 
age data registered in the image holding unit 218 and 
the respective sensory patterns using the aforemen- 
tioned image feature amount holding unit 222 and image 
feature amount/sensory pattern correspondence hold- 
ing unit 223, and registers them in the sensory pattern 
holding unit 220 (to be described later in step S87 in Fig. 
22). 

[0072] The processes executed in the image search 
apparatus of this embodiment with the above arrange- 
ment will be described below. 

[0073] Fig. 1 8 is a flow chart showing the operation of 
the image search apparatus of this embodiment. Refer- 
ring to Fig. 18, step SI is a processing module that im- 
plements the search request input process 201 in Fig. 
2, and inputs a search result. Note that the details of this 
process will be explained in detail later with reference 
to Fig. 19. 

[0074] Step S2 is a processing module that imple- 
ments the weight discrimination process 206^ and if it is 
determined with reference to the contents stored in the 
search weight holding unit 204 in the search request in- 
put process 201 in step SI that search weights are des- 
ignated, the designated values are stored in the deter- 
mined weight holding unit 207. On the other hand, if no 
search weights are designated, index words 80 are 
searched for a query word stored in the query word hold- 
ing unit 202 with reference to the concept discrimination 
dictionary 205 (Fig. 8) so as to read out a corresponding 
associated weight 83 and sensory pattern weight 84, 
and the readout weights are stored in the determined 
weight holding unit 207. If there is no index word 80 that 
is relevant to the contents of the query word holding unit 
202, a default value "5" is stored as both the associated 
and sensory pattern weights in the determined weight 
holding unit 207. 

[0075] It is checked with reference to the determined 
weight holding unit 207 in step S3 if the associated 
weight is zero. If the associated weight is zero, the flow 
advances to step S5; otherwise, the process in step S4 
is executed. Step S4 is a processing module that imple- 
ments the associative word unfolding process 208 and 
image content word search process 210 using associa- 
tive words in Fig. 2, and this process will be described 
in detail later with reference to Fig. 20. 
[0076] It is checked with reference to the determined 
weight holding unit 207 in step S5 if the sensory pattern 
weight is zero. If the sensory pattern weight is zero, 
since a search using sensory pattern is unnecessary, 
the flow advances to step S7; othenwise, the process in 
step S6 is executed. Step S6 is a processing module 
that implements the sensory pattern unfolding process 
212 and sensory pattern search process 214 in Fig. 2, 
and will be described in detail later with reference to Fig. 
21. 

[0077] Step S7 is a processing module that imple- 
ments the search result integration process 217 in Fig. 



2, and will be described in detail later with reference to 

Fig. 22. 

[0078] In step S8, image files corresponding to image 
IDs stored in the search result holding unit 21 6 as search 
s results obtained in step S7 are read out from the image 
holding unit 21 8, and are displayed. Note that the search 
result display process in step S8 is a known one which 
is prevalent in image search apparatuses of the same 
type. 

10 [0079] Fig. 19 is a flow chart showing the details of 
the search request input process 201 (step SI in Fig. 
18). In step S21, a query word serving as a search re- 
quest is input. The query word input is attained by stor- 
ing a character code input at the keyboard 1 5 in the que- 

is ry word holding unit 202 on the RAM 1 3. 

[0080] In step 822, using the query word stored in the 
query word holding unit 202 as a search request, search 
perspectives that are relevant to the search request are 
extracted from the concept discrimination dictionary 

20 205. In this case, all search perspectives corresponding 
to index words 80 (Fig. 8), which match the query word 
in the query word holding unit 202, are extracted. For 
example, when the query word is "mild", three search 
perspectives "color tone", "taste", and "general atmos- 

25 phere" can be obtained. 

[0081] It is checked in step S23 if a search perspective 
or perspectives is or are found by search perspective 
extraction in step S22. If a search perspective or per- 
spectives is or are found, the flow advances to step S24; 

30 otherwise, the flow advances to step S26. 

[0082] If search perspectives are found in step S22, 
they are displayed together with the query word, as in 
shown in Fig. 3, in step S24. In step S25, the user selects 
a desired one of the displayed search perspectives us- 

35 ing the user interface that has been described previous- 
ly with reference to Fig. 3. The selected search perspec- 
tive is stored in the search perspective holding unit 203. 
[0083] In step S26, the user inputs search weights 
which determine the weight balance on a search using 

40 associative words and a search using sensory pattern 
in actual search in relation to the search process method 
in response to the search request. In this embodiment, 
the user sets the weights using the user interface shown 
in Fig. 4. That is, the user operates the slide bar shown 

45 in Fig. 4 to designate the weight ratios (desired search 
weights) on associative words and sensory patterns by 
the length of the horizontal bar of the slider (the position 
of the button 41 ). When the user does not designate any 
search weights, he or she designates use of the default 

50 values of the weights shown in Fig. 4. 

[0084] It is checked in step S27 if search weights are 
designated. If it is instructed to use the default weight 
values, the processing ends. On the other hand, if 
search weights are designated, the flow advances to 

55 step S28 to store the designated associative word and 
sensory pattern weights designated in step 826 in the 
search weight holding unit 204, thus ending the process- 
ing. 
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[0085] Fig. 20 is a flow chart showing the details of 
the associative word unfolding process 208 and the im- 
age content word search process 210 using associative 
words (step S4 in Fig. 18). 

[0086] Referring to Fig. 20, the associative word dic- 
tionary 211 is searched using the query word stored in 
the query word holding unit 202 to obtain associative 
word data in step S41 . More specifically, the associative 
word dictionary 211 is searched for index words 91 (Fig. 
9), which match the query word, and registered associ- 
ative word data are extracted. If index words 91 that 
match the query word are found, all their associative I Ds 
are stored in the unfolded associative word holding unit 
209. 

[0087] In step S42, the concept discrimination diction- 
ary 205 is searched, and if an index word that matches 
the query word in the query word holding unit 202 is 
found, a search perspective 81 (Fig. 8) corresponding 
to that index word is extracted. The extracted search 
perspective is compared with that stored in the search 
perspective holding unit 203 in step S25, and if they 
match, an antithetic concept 82 corresponding to this 
index word is extracted. On the other hand, if the two 
search perspectives do not match, data in which the 
query word matches an index word continues to be 
searched for. Upon completion of checking for all the 
extracted search perspectives, the flow advances to 
step S43. 

[0088] In step S43, the associative word dictionary 
211 is searched for associative words having an index 
word, which matches the antithetic concept found in 
step S42. If an index word that match the antithetic con- 
cept is found, their associative IDs are stored in the un- 
folded associative word holding unit 209 by appending 
a status code indicating an antithetic concept thereto. 
[0089] In step S44, associative words are extracted 
based on the associative IDs stored in the unfolded as- 
sociative word holding unit 209, and the image content 
word holding unit 219 is searched for image content 
words that match the associative words. The search re- 
sults are stored in the search result holding unit 216. 
[0090] In this process, the associative IDs stored in 
the unfolded associative word holding unit 209 are ex- 
tracted, and corresponding associative data are extract- 
ed with reference to the associative word dictionary 211. 
Next, the association strengths 94 of the extracted as- 
sociative data are extracted, and are set in a work mem- 
ory ASCF (not shown) on the RAM 1 3. In this case, if a 
status code indicating an antithetic concept is appended 
to a given associative ID extracted from the unfolded 
associative word holding unit 209, the sign of the asso- 
ciation strength is inverted to indicate a negative asso- 
ciation strength. However, if the association strength is 
already a negative value, that associative data is dis- 
carded, and the next associative data is checked. 
[0091] Then, an associative perspective correspond- 
ing to each associative ID is extracted, and is compared 
with that stored in the search perspective holding unit 



203. If the two perspectives match, a predetermined val- 
ue a is set in a work memory VPF (not shown) assured 
on the RAM 1 3. If they do not match, a value a X 0. 1 is 
set in the work memory VPF. 

s [0092] Finally, the image content word holding unit 
21 9 is searched for image content words that match as- 
sociative words corresponding to the associative IDs. If 
an image content word is found, an image ID corre- 
sponding to that image content word is acquired from 

10 the image ID 72 (Fig. 7), and is set in the found image 
ID 100 (Fig. 10) in the search result holding unit 216. "1" 
is set in the number 101 of matched associative words, 
and the found associative ID is set in the associative 
word ID 102. Then, a value obtained by multiplying the 

15 value in the work memories ASCF and VPF on the RAM 
13 by a predetermined score p based on associative 
word matching is stored as an associative matching lev- 
el in the associative matching level 103. If an identical 
image ID has already been stored, the value of the 

20 number 101 of matched associative words is increment- 
ed by 1 , a new associative word ID is added to the as- 
sociative word ID 102, and the calculated associative 
matching level is added to the stored associative match- 
ing level 103 to update its value. 

25 [0093] Fig. 21 is a flow chart showing the details of 
the sensory pattern unfolding process 212, sensory pat- 
tern search process 214, and search result integration 
process 217. 

[0094] As described above, the user inputs a search 
30 request for searching images by the search request in- 
put process 201 . The search request contains one or a 
plurality of query words, search perspectives, and the 
like. The query word input in this embodiment is an ab- 
stract image word that expresses impressions of images 
35 such as "refreshing", "warm", and the like. In this em- 
bodiment, assume that an image word "refreshing" is 
stored. 

[0095] Steps S61 and S62 are implemented by the 

sensory pattern unfolding process 212. In step S61 ., the 

40 image word held in the query word holding unit 202 is 
unfolded into sensory patterns with reference to the im- 
age word/sensory pattern correspondence holding unit 
215. In this embodiment, the query word holding unit 
202 stores the image word "refreshing", the unfolded as- 

45 sociative word holding unit 209 holds associative words 
"forest", "tableland", "blue sky", and the like unfolded 
from "refreshing", and the image word is unfolded into 
corresponding sensory pattern IDs with reference to the 
image word/sensory pattern correspondence holding 

50 unit 215. For example, sensory pattern IDs "005" and 
"006" corresponding to image word "refreshing" - asso- 
ciative word "forest" are acquired, and a sensory pattern 
ID "007" corresponding to image word "refreshing" - as- 
sociative word "tableland" is acquired. 

55 [0096] The flow then advances to step 862 to store 
the sets of unfolded sensory pattern IDs and image 
words/associative words in the unfolded sensory pat- 
tern holding unit 213. The data storage in the unfolded 
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sensory pattern holding unit 21 3 is as has already been 

described previously with reference to Fig. 11. 

[0097] The flow advances to step S63. Steps S63 and 

564 are implemented by the sensory pattern search 
process 21 4. In step S63, all image IDs of images having 
matching levels larger than zero with respect to the sen- 
sory pattern IDs stored in the unfolded sensory pattern 
holding unit 213 are acquired. This process is done for 
all the sensory patterns held in the unfolded sensory pat- 
tern holding unit 213. Note that the sensory pattern 
search process 214 acquires image IDs having match- 
ing levels larger than zero with respect to the sensory 
pattern IDs respectively unfolded from the query word 
and antithetic concept. 

[0098] In step S64, sets of acquired sensory pattern 

IDs, image IDs, and their matching levels acquired in 
step S63 are stored in the search result holding unit 21 6. 
[0099] The flow then advances to step S65. Steps 

565 to S69 are implemented by the search result inte- 
gration process 217. That is, two sets of search results, 
i.e., the image content word search results using asso- 
ciative words and sensory pattern search results, which 
are held in the search result holding unit 216; are inte- 
grated into one set of search results on the basis of the 
search weights stored in the determined weight holding 
unit 207 with reference to those search results. When 
the sensory pattern search results include a sensory 
pattern based on the antithetic concept to the query 
word, the corresponding image is excluded from the in- 
tegrated results. Or the sensory pattern matching level 
of an image including an sensory pattern of the antithetic 
concept may be lowered upon integration. 

[0100] In this integration process, a method of obtain- 
ing common elements of two sets of search results in 
units of associative words (ANDing search results), a 
method of calculating integrated matching levels based 
on the weights on the searches, and selecting appropri- 
ate search results in descending order of integrated 
matching levels, and the like are available. In this em- 
bodiment, the method of calculating the integrated 
matching levels will be exemplified below. 
[0101] Let A be the associative matching level of an 
image that matches, e.g., an associative word "forest" 
stored in the search result holding unit 216, B be the 
sensory matching level of an image that matches the 
sensory pattern ID "005" corresponding to the associa- 
tive word "forest", and w1 and w2 (w1 + w2 = 1) be the 
search weights stored in the determined weight holding 
unit 207. Then, the integrated matching level is given by: 

Integrated matching level = w1 X A + w2 X B 

or 

2 2 1/2 

Integrated matching level = (w1 X A + w2 X B ) 



[0102] The integrated matching levels of all sensory 
patterns of all associative words are calculated. When 
one image ID has matching levels larger than zero with 
respect to a plurality of sensory pattern IDs, a plurality 
s of integrated matching levels are obtained for one im- 
age. However, in this case, an image with the highest 
integrated matching level is adopted as a search result 
(step S65). 

[0103] This process is done for all images corre- 
ct? spending to either set of search results larger than zero, 
and images whose integrated matching levels are larger 
than a predetermined threshold value X are selected as 
integrated search results (steps S66, S67, and S68). 
[0104] In step 869, the sets of image IDs and their 
15 integrated matching levels are stored in the search re- 
sult holding unit 216, thus ending the search process. 
[0105] An image registration process for registering 
test images will be explained below with reference to 
Fig. 22. 

20 [0106] Fig. 22 is a flow chart showing an example of 
a search pre-process executed upon registering imag- 
es. This process is controlled in accordance with a 
processing program stored in the DISK 14. 
[0107] In step S81, the user designates an image to 
25 be registered. The image to be registered is designated 
from those stored in an external storage device, an im- 
age input device, an image database server connected 
to this image processing apparatus, or the like (none of 
them are shown). In this embodiment, assume that im- 
30 ages serving as test images are stored in advance, and 
the image to be registered is selected from them. 
[0108] The flow then advances to step S82. In step 
882, an image ID corresponding to the designated im- 
age file name, and various kinds of image information 
55 required for registration are acquired, and are supplied 
to the image feature extraction process 221 . The image 
ID is stored in correspondence with the image file name 
to manage an image, and is acquired by searching data 
in the image holding unit 218 using the image file name. 
40 Various kinds of information of the image include pixel 
values indicating the width and height of an image, the 
number of bits per pixel, the image size (in units of 
bytes), the address of the area where a bitmap image 
is actually stored, and the like, for example, when the 
45 file format of this image is the bitmap format. Since these 
pieces of image information are stored in the header 
field of the image file, they can be acquired by referring 
to the header field. Even when the file format of the im- 
age is not the bitmap format but JFIF or FlashPix, re- 
50 quired information can be similarly obtained from the 
header field of a file. Or the image holding unit 21 8 may 
store such image information, and the image informa- 
tion may be acquired by referring to the image holding 
unit 218 upon registration. 
55 [0109] The flow advances to step S83. Step 883 is 
implemented by the image feature amount extraction 
process 221, and extracts physical image feature 
amounts by analyzing the image information corre- 
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spending to the designated image ID. Fig. 15 above 
shows an example of the image feature amounts in this 
embodiment, and representative colors are extracted in 
units of image regions/blocks. The representative color 
may be obtained by using a scheme of analyzing an ac- 
tual bitmap image using various kinds of input image in- 
formation in units of pixels, and calculating the average 
value of color components (values in a color space such 
as RGB, HVC, or the like) used in each region or block, 
or a color component with the highest frequency of oc- 
currence as a representative color. 
[0110] The flow advances to step S84. In step S84, 
image feature amounts c1 to on extracted in step S83 
are stored in the image feature amount holding unit 222 
in correspondence with the image ID of this image. The 
data storage format in this case is as has already been 
described previously with reference to Fig. 16. 
[0111] The flow advances to step S85, and all sensory 
pattern IDs stored in the image feature amount/sensory 
pattern correspondence holding unit 223, and image 
feature amounts corresponding to those sensory pat- 
terns are acquired with reference to the image feature 
amount/sensory pattern correspondence holding unit 
223. In this embodiment, the chromatic feature amounts 
correspond to the individual sensory patterns, as has 
already been described previously with reference to Fig. 
17. 

[0112] The flow advances to step S86, and the match- 
ing level between each sensory pattern acquired in step 
S85 and the image feature amounts corresponding to 
this image is calculated. This process is done by the 
sensory pattern determination process 224. That is, the 
chromatic image feature amounts corresponding to 
each of the sensory patterns acquired in step S85 are 
compared with the image feature amounts extracted in 
step S83to calculate their matching level. The matching 
levels for all sensory patterns stored in the image feature 
amount/sensory pattern correspondence holding unit 
223 are calculated. The matching level is calculated us- 
ing a scheme such as vector computations, statistic 
processes, or the like using cosine measure. 
[0113] The flow advances to step S87. In step S87, 
the matching levels between all the sensory patterns 
and the image calculated in step S86 are stored in the 
sensory pattern holding unit 220 in correspondence with 
the image ID of this image. The image storage example 
in the sensory pattern holding unit 220 is as has already 
been described previously with reference to Fig. 13. 
[0114] The aforementioned process is done for all im- 
ages to be registered. 

[0115] As described above, according to this embod- 
iment, a search using feature amount data of multimedia 
information itself and a search using a content word ap- 
pended to multimedia information are made on the basis 
of associative words, which are associated with a query 
word, and final search results can be obtained from the 
results of the two search processes. Forthis reason, de- 
sired image information can be accurately extracted. 



[Oil 6] As described above, according to this embod- 
iment, since a search request (query word, search per- 
spective, and the like) which is to be considered upon 
searching for desired multimedia information can be 
s designated, an appropriate search can be made in ac- 
cordance with the designated search request, and de- 
sired image information can be accurately extracted. 
[011 7] According to this embodiment, upon obtaining 
search results by integrating search results obtained by 
10 a search using keywords appended to images, and 
those obtained by a search using feature amount data 
of images themselves, since the weight ratios on the two 
search processes can be changed in correspondence 
with a query word, the image information wanted can be 
15 accurately extracted. For example, when a keyword 
"happy" is input as a search request, it is hard to asso- 
ciate it with image feature amounts since its meaning is 
essentially lexical. Hence, if a search that attaches im- 
portance on the image feature amount is made, images 
20 which do not match the search request are highly likely 
to be presented. On the other hand, for example, when 
a keyword "showy" is input as a search request, the key- 
word "showy" is more likely to evoke meanings meas- 
urable as image feature amounts. For this reason, if a 
25 search is made while attaching importance on content 
words appended to images, images which do not match 
the search request indicated by the input keyword are 
highly likely to be presented. Or actually "showy" images 
may be excluded from the search results. By contrast, 
30 according to this embodiment, when a query word "hap- 
py" is set via the user interface shown in Fig. 4, heavier 
weights are set on associative words; when a query 
word "showy" is set, heavier weights are set on sensory 
patterns, thus making an accurate search with respect 
55 to either query word. Of course, when the associated 
weight 83 and sensory pattern weight 84 in the concept 
discrimination dictionary 205 are appropriately set, an 
appropriate search can be made by only instructing to 
"use default weight values" on the user interface shown 
40 in Fig. 4. 

[0118] In the above embodiment, image information 
is used as stored information serving as test images. As 
for multimedia information (e.g., audio information) oth- 
er than image information, the present invention can be 
45 applied by executing information feature amount extrac- 
tion, and pairing the extracted information feature 
amount with sensory patterns. 

[011 9] In the above description, the image holding unit 
218, image content word holding unit 219, and sensory 
50 pattern holding unit 220 which undergo a search are al- 
located on the DISK 14 that builds a single device, but 
these building components may be distributed on differ- 
ent devices, and processes may be done on the network 
via the NIC 19. 

55 [01 20] Note that the present invention may be applied 
to either a system constituted by a plurality of devices 
(e.g., a host computer, an interface device, a reader, a 
printer and the like), or an apparatus consisting of asin- 
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gle equipment (e.g., a copying machine, a facsimile ap- 
paratus, or tine like). 

[0121] The objects of the present invention are also 
achieved by supplying a storage medium, which records 
a program code of a software program that can imple- 
ment the functions of the above-mentioned embodi- 
ments to the system or apparatus, and reading out and 
executing the program code stored in the storage me- 
dium by a computer (or a CPU or MPU) of the system 
or apparatus. 

[0122] In this case, the program code itself read out 
from the storage medium implements the functions of 
the above-mentioned embodiments, and the storage 
medium which stores the program code constitutes the 
present invention. 

[0123] As the storage medium for supplying the pro- 
gram code, for example, a floppy disk, hard disk, optical 
disk, magneto-optical disk, CD-ROM, CD-R, magnetic 
tape, nonvolatile memory card, ROM, and the like may 
be used. 

[01 24] The functions of the above-mentioned embod- 
iments may be implemented not only by executing the 
readout program code by the computer but also by some 
or all of actual processing operations executed by an 
OS (operating system) running on the computer on the 
basis of an instruction of the program code. 
[0125] Furthermore, the functions of the above-men- 
tioned embodiments may be implemented by some or 
all of actual processing operations executed by a GPU 
or the like arranged in a function extension board or a 
function extension unit^ which is inserted in or connected 
to the computer, after the program code read out from 
the storage medium is written in a memory of the exten- 
sion board or unit. 

[0126] As many apparently widely different embodi- 
ments of the present invention can be made without de- 
parting from the spirit and scope thereof, it is to be un- 
derstood that the invention is not limited to the specific 
embodiments thereof except as defined in the append- 
ed claims. 



Claims 

1. An information search apparatus for searching in- 
formation based on an input query word, compris- 
ing: 

first search means for determining a query key- 
word on the basis of the query word, and 
searching information on the basis of the query 
keyword; 

second search means for determining a feature 

amount corresponding to the query word, and 
searching information on the basis of the fea- 
ture amount; 

setting means for setting a search weight to be 
assigned to search results of said first and sec- 



ond search means; and 
integration means for integrating search results 
obtained by said first and second search means 
in accordance with the search weight set by 
s said setting means. 

2. The apparatus according to claim 1, wherein the 
search weight includes a first weight corresponding 
to the search result of said first search means, and 

10 a second weight corresponding to the search result 
of said second search means, and 

said integration means applies the first weight 
to a search matching level of each information as 
the search result of said first search means and the 

15 second weight to a search matching level of each 
information as the search result of said second 
search means to obtain an integrated search 
matching level, and obtains integrated search re- 
sults on the basis of the integrated search matching 

20 level. 

3. The apparatus according to claim 2, wherein said 
integration means selects a predetermined number 
of pieces of information in descending order of in- 

25 tegrated search matching level, and determines the 
selected information as the integrated search re- 
sults. 

4. The apparatus according to claim 1 , wherein said 
30 setting means allows a user to set desired weight 

ratios with respect to the search results of said first 
and second search means. 

5. The apparatus according to claim 1, further com- 
35 prising: 

a weight dictionary which registers weights cor- 
responding to said first and second search 
means in relation to the query word, and 
40 wherein said setting means sets the weights 

with reference to said weight dictionary. 

6. The apparatus according to claim 5, wherein said 

first search means derives an associative word as- 
45 sociated with the query word, and uses the query 
word and the derived associative word as query 
keywords, 

said weight dictionary registers weights in units 
50 of associative perspectives that connect query 

words and associative words in units of query 
words, and 

said setting means sets the weights with refer- 
55 ence to said weight dictionary on the basis of 

the query word and an associative perspective 
designated by a user. 
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7. The apparatus according to claim 1 , wherein the in- 
tormation searched is innage data, 

said apparatus further comprises: 

an image content word holding unit for storing 
the Image data and content words which ver- 
balize concepts expressed in the image data in 
correspondence with each other; and 
an associative word dictionary for storing asso- 
ciative words associated with the content 
words, and 

said first search means acquires an associative 
word corresponding to the query word from said 
associative word dictionary, and searches said 
image content word holding unit on the basis of 
the acquired associative word. 

8. The apparatus according to claim 7, further com- 
prising: 

a concept discrimination dictionary for storing 
index words and antithetic concepts corre- 
sponding to the index words in correspondence 
with each other; and 

input means for inputting the query word and a 
search perspective, and 
wherein said first search means acquires an in- 
dex word and antithetic concept corresponding 
to the query word from said concept discrimi- 
nation dictionary on the basis of the query word 
and search perspective input by said input 
means, and acquires an associative word cor- 
responding to the query word from said asso- 
ciative word dictionary on the basis of the ac- 
quired index word and antithetic concept. 

9. The apparatus according to claim 1, further com- 
prising: 

a holding unit for storing associative words and 
sensory patterns in correspondence with each 
other, and 

wherein said second search means acquires a 
sensory pattern corresponding to an associa- 
tive word, which corresponds tothe query word, 
from said holding unit, and extracts a feature 
amount of the acquired sensory pattern as the 
feature amount corresponding to the query 
word. 

10. The apparatus according to claim 1 , wherein multi- 
media information is image information, and the 
feature amount is a physical image feature amount 
obtained by analyzing the image information. 

11. The apparatus according to claim 10, wherein the 
feature amount includes at least one of color 
scheme information, composition information, and 



shape information contained of an image. 

12. An information search method for searching infor- 
mation based on an input query word, comprising: 

5 

the first search step of determining a query key- 
word on the basis of the query word, and 
searching information on the basis of the query 
keyword; 

10 the second search step of determining a feature 

amount corresponding to the query word, and 
searching information on the basis of the fea- 
ture amount; 

the setting step of setting a search weight to be 
15 assigned to search results in the first and sec- 

ond search steps: and 

the integration step of integrating search re- 
sults obtained in the first and second search 
steps in accordance with the search weight set 
20 in the setting step. 

13. The method according to claim 12, wherein the 
search weight includes a first weight corresponding 
to the search result in the first search step, and a 

25 second weight corresponding to the search result 
in the second search step, and 

the integration step includes the step of ap- 
plying the first weight to a search matching level of 
each information as the search result in the first 

30 search step and the second weight to a search 
matching level of each information as the search re- 
sult in the second search step to obtain an integrat- 
ed search matching level, and obtaining integrated 
search results on the basis of the integrated search 

35 matching level. 

14. The method according to claim 13, wherein the in- 
tegration step includes the step of selecting a pre- 
determined number of pieces of information in de- 

40 scending order of integrated search matching level, 
and determining the selected information as the in- 
tegrated search results. 

15. The method according to claim 1 2, wherein the set- 
45 ting step includes the step of allowing a user to set 

desired weight ratios with respect to the search re- 
sults in the first and second search steps. 

16. The method according to claim 12, further compris- 
50 ing: 

a weight dictionary which registers weights cor- 
responding to the first and second search steps 
in relation to the query word, and 
55 wherein the setting step includes the step of 

setting the weights with reference to said 
weight dictionary. 
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17. The method according to claim 1 6, wherein the first 
search step includes the step of deriving an asso- 
ciative word associated with the query word; and 
using the query word and the derived associative 
word as query keywords, 

said weight dictionary registers weights in units 
of associative perspectives that connect query 
words and associative words in units of query 
words, and 

the setting step includes the step of setting the 
weights with reference to said weight dictionary 
on the basis of the query word and an associ- 
ative perspective designated by a user. 

18. The method according to claim 12, wherein the in- 
formation searched is image data, 

said method further comprises: 

an image content word holding unit for storing 
the image data and content words which ver- 
balize concepts expressed in the image data in 
correspondence with each other; and 
an associative word dictionary for storing asso- 
ciative words associated with the content 
words, and 

the first search step includes the step of acquir- 
ing an associative word corresponding to the 
query word from said associative word diction- 
ary, and searching said image content word 
holding unit on the basis of the acquired asso- 
ciative word. 

19. The method according to claim 18, further compris- 
ing: 

a concept discrimination dictionary for storing 
index words and antithetic concepts corre- 
sponding to the index words in correspondence 

with each other; and 

the input step of inputting the query word and 
a search perspective, and 
wherein the first search step includes the step 
of acquiring an index word and antithetic con- 
cept corresponding to the query word from said 
concept discrimination dictionary on the basis 
of the query word and search perspective input 
in the input step, and acquiring an associative 
word corresponding to the query word from said 
associative word dictionary on the basis of the 
acquired index word and antithetic concept. 

20. The method according to claim 1 2, further compris- 
ing: 

a holding unit for storing associative words and 
sensory patterns in correspondence with each 
other, and 



a code of the first search step of determining a 
query keyword on the basis of the query word, 
and searching information on the basis of the 
query keyword; 

a code of the second search step of determin- 
ing a feature amount corresponding to the que- 
ry word, and searching information on the basis 
of the feature amount; 

a code of the setting step of setting a search 
weight to be assigned to search results in the 
first and second search steps; and 
a code of the integration step of integrating 
search results obtained in the first and second 
search steps in accordance with the search 
weight set in the setting step. 

40 24. A method of searching a database on the basis of 
more than one search criteria, including the step of 
allocating relative weight to each of the criteria to 
be used in the search. 

45 25. A method as claimed in claim 24 wherein one of the 
criteria is a search word. 

26. A method as claimed in claim 25 wherein another 
one of the criteria is a parameter associated with 

50 image content of the searched objects. 

27. A storage medium storing data in a form which is 
searchable using a method as claimed in any of 
claims 12 to 22. 

55 

28. A signal for transmission over a communications 
channel comprising computer implementable in- 
structions for carrying out a method as claimed in 



wherein the second search step includes the 
step of acquiring a sensory pattern correspond- 
ing to an associative word, which corresponds 
to the query word, from said holding unit, and 
5 extracting a feature amount of the acquired 

sensory pattern as the feature amount corre- 
sponding to the query word. 

21. The method according to claim 12, wherein multi- 
10 media information is image information, and the 

feature amount is a physical image feature amount 
obtained by analyzing the image information. 

22. The method according to claim 21 , wherein the fea- 
15 ture amount includes at least one of color scheme 

information, composition information, and shape in- 
formation contained of an image. 

23. A storage medium for storing a control program 
20 which makes a computer search information based 

on an input query word, said control program com- 
prising: 
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